Support creating mashes from multiple tags by clefrks · Pull Request #10 · clearlinux/koji-setup-scripts

clefrks · 2019-10-09T05:07:20Z

Currently koji-setup-scripts only cater 1 tag for snapshot.
Added feature for multiple tag support.

Also removing hardcoded value for dist- in bootstrap.sh and deploy-mash.sh

bryteise · 2019-10-09T14:48:15Z

koji-setup/mash.sh

 	mkdir -p "$MASH_DIR_NEW"
 	create_dist_repos "$MASH_DIR_NEW"
-	if [[ -e "$MASH_TRACKER_DIR" ]]; then
-		mv "$MASH_TRACKER_DIR" "$MASH_DIR_OLD"


Since $MASH_DIR_OLD needs to exist now as the move is moving it to a subdirectory rather than just renaming the directory to .old I think this needs a mkdir -p $MASH_DIR_OLD.

ah yes! i will update that and $MASH_TRACKER_DIR!
thanks!

bryteise · 2019-10-09T14:52:05Z

koji-setup/mash.sh

+		mv "$MASH_TRACKER_DIR/$KOJI_TAG" "$MASH_DIR_OLD/$KOJI_TAG"
 	fi
-	mv "$MASH_DIR_NEW" "$MASH_TRACKER_DIR"
+	mv "$MASH_DIR_NEW/$KOJI_TAG" "$MASH_TRACKER_DIR"


Similarly with $MASH_TRACKER_DIR being used as a subdir, the script will need to ensure it is created first.

gtkramer · 2019-10-24T19:58:59Z

@clefrks we've had a number of changes go in recently that remove our usage of the mash tool and make the replacement script faster. It was the last python2 dependency we had in the DevOps flow this repo creates. Would you be able to rebase please? It looks like you have some good changes that I'd like to get in.

Currently snapshot aka mash is using clear as default value for repo name. With this changes, it'll follow the TAG_NAME that users set in parameter.sh to create mash for the tags. Signed-off-by: candrew <[email protected]>

Instead of hardcoding dist-, just add into $TAG_NAME in parameter.sh Signed-off-by: candrew <[email protected]>

Ensure the creation of the folders before moving the collaterals into the folders. Signed-off-by: candrew <[email protected]>

clefrks · 2019-10-25T06:44:47Z

@gtkramer i've done rebasing, please review thanks!

gtkramer · 2019-10-25T20:35:44Z

koji-setup/deploy-mash.sh

 cp -f "$SCRIPT_DIR"/mash.sh "$MASH_SCRIPT_DIR"
 mkdir -p /etc/systemd/system
-cat > /etc/systemd/system/mash.service <<- EOF
+cat > /etc/systemd/system/mash@$TAG_NAME.service <<- EOF


I think for a case like this, we want the service file on disk to be named [email protected] to indicate to systemd that we want to pass a parameter, but not hard-code the parameter for all instantiations of the service.

gtkramer · 2019-10-25T20:38:35Z

koji-setup/mash.sh

 MASH_DIR="${MASH_DIR:-/srv/mash}"
-MASH_TRACKER_FILE="$MASH_DIR"/latest-mash-build
+MASH_TRACKER_FILE="$MASH_DIR"/latest-${KOJI_TAG}-build


If I am understanding the intent of the PR correctly, I think we want

MASH_DIR="${MASH_DIR:-"/srv/mash/$KOJI_TAG"}"

Then we wouldn't modify MASH_TRACKER_FILE, but rather, self-contain a similar structure to every mash in the name of a folder that varies by tag. This allows for more fine-grained management of each repo and allows multiple repos for each tag to build up independently, if so desired.

gtkramer · 2019-10-25T20:41:31Z

koji-setup/mash.sh

-	make_repo "${source_dir}" "${output_dir}" "clear/${BUILD_ARCH}/os" "Packages" "${bin_rpm_paths}" "${comps_file}" &
-	make_repo "${source_dir}" "${output_dir}" "clear/${BUILD_ARCH}/debug" "." "${debuginfo_rpm_paths}" &
-	make_repo "${source_dir}" "${output_dir}" "clear/source/SRPMS" "." "${src_rpm_paths}" &
+	make_repo "${source_dir}" "${output_dir}" "${KOJI_TAG}/${BUILD_ARCH}/os" "Packages" "${bin_rpm_paths}" "${comps_file}" &
+	make_repo "${source_dir}" "${output_dir}" "${KOJI_TAG}/${BUILD_ARCH}/debug" "." "${debuginfo_rpm_paths}" &
+	make_repo "${source_dir}" "${output_dir}" "${KOJI_TAG}/source/SRPMS" "." "${src_rpm_paths}" &
 	wait

-	create_dnf_conf "${work_dir}/dnf-os.conf" "${output_dir}/clear/${BUILD_ARCH}/os" clear-os
-	create_dnf_conf "${work_dir}/dnf-debug.conf" "${output_dir}/clear/${BUILD_ARCH}/debug" clear-debug
-	create_dnf_conf "${work_dir}/dnf-SRPMS.conf" "${output_dir}/clear/source/SRPMS" clear-SRPMS
+	create_dnf_conf "${work_dir}/dnf-os.conf" "${output_dir}/${KOJI_TAG}/${BUILD_ARCH}/os" clear-os
+	create_dnf_conf "${work_dir}/dnf-debug.conf" "${output_dir}/${KOJI_TAG}/${BUILD_ARCH}/debug" clear-debug
+	create_dnf_conf "${work_dir}/dnf-SRPMS.conf" "${output_dir}/${KOJI_TAG}/source/SRPMS" clear-SRPMS

-	write_packages_file "${work_dir}/dnf-os.conf" "$output_dir/clear/$BUILD_ARCH/packages-os"
-	write_packages_file "${work_dir}/dnf-debug.conf" "$output_dir/clear/$BUILD_ARCH/packages-debug"
-	write_packages_file "${work_dir}/dnf-SRPMS.conf" "$output_dir/clear/source/packages-SRPMS"
+	write_packages_file "${work_dir}/dnf-os.conf" "$output_dir/${KOJI_TAG}/$BUILD_ARCH/packages-os"
+	write_packages_file "${work_dir}/dnf-debug.conf" "$output_dir/${KOJI_TAG}/$BUILD_ARCH/packages-debug"
+	write_packages_file "${work_dir}/dnf-SRPMS.conf" "$output_dir/${KOJI_TAG}/source/packages-SRPMS"


For these function calls, we unfortunately need to keep the clear/ structure. For better or worse, a lot of our processes today depend on this naming convention. I don't think the change to these lines would be needed if we contain the mash structure in a root folder name that varies by tag.

gtkramer · 2019-10-25T20:47:33Z

koji-setup/mash.sh

 set -e
 . /etc/profile.d/proxy.sh || :

+KOJI_TAG="${KOJI_TAG:-"$1"}"


"${1:?"Missing tag parameter"}"

Is probably a more accurate parameter expansion here. If the service is enabled without a parameter, we want to indicate that this is required and have the script fail.

gtkramer · 2019-10-25T20:53:08Z

koji-setup/mash.sh

-	if [[ -e "$MASH_TRACKER_DIR" ]]; then
-		mv "$MASH_TRACKER_DIR" "$MASH_DIR_OLD"
+	if [[ -e "$MASH_TRACKER_DIR/$KOJI_TAG" ]]; then
+		mkdir -p "$MASH_DIR_OLD"
+		mv "$MASH_TRACKER_DIR/$KOJI_TAG" "$MASH_DIR_OLD/$KOJI_TAG"
 	fi
-	mv "$MASH_DIR_NEW" "$MASH_TRACKER_DIR"
+    mkdir -p "$MASH_TRACKER_DIR"
+	mv "$MASH_DIR_NEW/$KOJI_TAG" "$MASH_TRACKER_DIR"
 	rm -rf "$MASH_DIR_OLD"
+    rm -rf "$MASH_DIR_NEW"


I think with the change to the root folder, none of these changes are needed?

gtkramer · 2019-10-25T21:01:35Z

Moving the change to support multiple tags to more of a root directory for each mash structure, we would also need to modify parameters.sh. The following changes would need made:

export TAG_NAME=dist-clear
export MASH_DIR="/srv/mash/$TAG_NAME"

To mirror what is in mash.sh.

gtkramer · 2019-10-25T21:01:44Z

@clefrks Would these suggestions work for you?

bryteise reviewed Oct 9, 2019

View reviewed changes

clefrks requested a review from bryteise October 17, 2019 07:11

clefrks added 3 commits October 25, 2019 14:41

Include support for multiple snapshot repo

ecdb82a

Currently snapshot aka mash is using clear as default value for repo name. With this changes, it'll follow the TAG_NAME that users set in parameter.sh to create mash for the tags. Signed-off-by: candrew <[email protected]>

Remove hardcoded dist- value

4b06ff5

Instead of hardcoding dist-, just add into $TAG_NAME in parameter.sh Signed-off-by: candrew <[email protected]>

Add $MASH_DIR_OLD and $MASH_TRACKER_DIR folder

fa8efd4

Ensure the creation of the folders before moving the collaterals into the folders. Signed-off-by: candrew <[email protected]>

clefrks force-pushed the candrew branch from 0d216b7 to fa8efd4 Compare October 25, 2019 06:42

gtkramer reviewed Oct 25, 2019

View reviewed changes

gtkramer changed the title ~~Support multiple snapshot(mash)~~ Support creating mashes from multiple tags Oct 25, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support creating mashes from multiple tags#10

Support creating mashes from multiple tags#10
clefrks wants to merge 3 commits intoclearlinux:masterfrom
clefrks:candrew

clefrks commented Oct 9, 2019

Uh oh!

bryteise Oct 9, 2019

Uh oh!

clefrks Oct 14, 2019

Uh oh!

bryteise Oct 9, 2019

Uh oh!

gtkramer commented Oct 24, 2019 •

edited

Loading

Uh oh!

clefrks commented Oct 25, 2019

Uh oh!

gtkramer Oct 25, 2019

Uh oh!

gtkramer Oct 25, 2019 •

edited

Loading

Uh oh!

gtkramer Oct 25, 2019 •

edited

Loading

Uh oh!

gtkramer Oct 25, 2019

Uh oh!

gtkramer Oct 25, 2019

Uh oh!

gtkramer commented Oct 25, 2019 •

edited

Loading

Uh oh!

gtkramer commented Oct 25, 2019 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

clefrks commented Oct 9, 2019

Uh oh!

bryteise Oct 9, 2019

Choose a reason for hiding this comment

Uh oh!

clefrks Oct 14, 2019

Choose a reason for hiding this comment

Uh oh!

bryteise Oct 9, 2019

Choose a reason for hiding this comment

Uh oh!

gtkramer commented Oct 24, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clefrks commented Oct 25, 2019

Uh oh!

gtkramer Oct 25, 2019

Choose a reason for hiding this comment

Uh oh!

gtkramer Oct 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gtkramer Oct 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gtkramer Oct 25, 2019

Choose a reason for hiding this comment

Uh oh!

gtkramer Oct 25, 2019

Choose a reason for hiding this comment

Uh oh!

gtkramer commented Oct 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gtkramer commented Oct 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gtkramer commented Oct 24, 2019 •

edited

Loading

gtkramer Oct 25, 2019 •

edited

Loading

gtkramer Oct 25, 2019 •

edited

Loading

gtkramer commented Oct 25, 2019 •

edited

Loading

gtkramer commented Oct 25, 2019 •

edited

Loading